An artificial neural network, or simply “neural network,” is a computer model, resembling a biological network of neurons. Neural networks are a family of methods within machine learning, under artificial intelligence. Neural networks typically have a plurality of artificial “neurons” that are connected by artificial “synapses.” There are many types of NNs that have various arrangements of neurons including partially or fully connected NNs, recurrent NNs (RNNs), convolutional NNs (CNNs), etc. For example, a connected NN typically has an input later, multiple middle or hidden layer(s), and an output layer, where each layer has a plurality (e.g., 100s to 1000s) of artificial neurons. In a fully-connected NN, each neuron in a layer (N) may be connected by an artificial “synapse” to all neurons in a prior (N−1) layer and subsequent (N+1) layer, whereas in a partially-connected NN only a subset of neurons in adjacent layers are connected.
A NN is trained based on a leaning dataset to solve or learn a weight of each synapse indicating the strength of that connection. The weights of the synapses are generally initialized, e.g., randomly. Training is performed by iteratively inputting a sample or set of samples from the training dataset into the NN, outputting a result of the NN applied to the dataset, calculating errors between the expected (e.g., target) and actual outputs, and adjusting NN weights to minimize errors. Training may be repeated until the error is minimized or converges. Typically, multiple passes (e.g., tens or hundreds) through the training set are performed (e.g., each sample is input into the NN multiple times). Each complete pass over the entire training dataset is referred to as one “epoch”.
Training an accurate NN is typically a complex and time-consuming task. A typical training process consists of billions (or trillions) of calculations that may take hours, days or even weeks to complete. So, once produced, other devices or parties may want to replicate an accurately trained NN, instead of developing one from scratch. However, NNs and their training data are often kept secret for data privacy, security, or proprietary reasons. Without access to the secret training dataset, other devices or parties cannot currently replicate a pre-trained NN.
Another limitation of conventional NNs is that new information cannot be added or old information deleted from the NN without retraining based on the entire training dataset. Training based solely on the changed data will override old neural connections and knowledge. Thus, to incorporate new data into the NN, no matter how minor, the training process is re-run from scratch with the original training dataset and the new samples added, which is a time-consuming process. Similarly, undesirable training data cannot be eliminated from the NN without completely retraining the NN based on the entire revised training dataset. In fact, any modification that re-trains the NN requires retraining over the entire training dataset. Thus, the NN cannot be modified or improved by devices or parties that do not have access to the original training dataset.
Accordingly, there is a need in the art to provide a mechanism to mimic and/or modify a NN without access to its secret original training dataset.